Skip to content

Feat CSV source#72

Merged
abnegate merged 17 commits into
mainfrom
feat-csv
Apr 16, 2025
Merged

Feat CSV source#72
abnegate merged 17 commits into
mainfrom
feat-csv

Conversation

@ItzNotABug
Copy link
Copy Markdown
Contributor

Adds the CSV source handler.

@ItzNotABug ItzNotABug requested a review from abnegate April 9, 2025 12:55
@ItzNotABug ItzNotABug self-assigned this Apr 9, 2025
Comment thread src/Migration/Sources/Csv.php Outdated
Comment thread src/Migration/Sources/Csv.php Outdated
Comment thread src/Migration/Sources/Csv.php Outdated
@ItzNotABug
Copy link
Copy Markdown
Contributor Author

@abnegate here's a quick benchmark -

  1. 3 documents in 0.0039 sec.
  2. 1,00,000 documents in 5.9281 sec
  3. 10,00,000 documents in 65.5917 sec
  4. 50,00,000 documents in 318.6609 sec


The controller froze on a CSV with 1,00,00,000 documents but I'll assume its way more than the last set.


@abnegate
If the insertion order doesn't matter, can we not use coroutines here so its not really sequential?

@ItzNotABug ItzNotABug requested a review from abnegate April 12, 2025 11:31
Comment thread src/Migration/Sources/Csv.php Outdated
Comment thread src/Migration/Sources/Csv.php Outdated
Comment thread src/Migration/Sources/Csv.php Outdated
Comment thread src/Migration/Sources/Csv.php Outdated
Comment thread src/Migration/Sources/Csv.php Outdated
Comment thread src/Migration/Sources/Csv.php Outdated
Comment thread src/Migration/Sources/Csv.php Outdated
Comment thread src/Migration/Sources/Csv.php Outdated
Comment thread src/Migration/Sources/Csv.php Outdated
Comment thread src/Migration/Sources/Csv.php Outdated
@ItzNotABug ItzNotABug requested a review from abnegate April 15, 2025 05:26
Comment thread src/Migration/Sources/Csv.php
Comment thread src/Migration/Sources/Csv.php Outdated
Comment thread src/Migration/Sources/Csv.php Outdated
Comment thread src/Migration/Sources/Csv.php Outdated
Comment thread src/Migration/Sources/CSV.php Outdated
Copy link
Copy Markdown
Member

@abnegate abnegate left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's add some tests as well

Comment thread src/Migration/Sources/CSV.php Outdated
Comment thread src/Migration/Sources/CSV.php Outdated
@ItzNotABug ItzNotABug requested a review from eldadfux April 16, 2025 07:51
@abnegate abnegate merged commit 545705e into main Apr 16, 2025
3 checks passed
@abnegate abnegate deleted the feat-csv branch April 16, 2025 07:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants